Skip to content

Add ibis/xorq analysis backend - #518

Closed
paddymul wants to merge 6 commits into
mainfrom
feat/paf-v2-ibis-v2
Closed

Add ibis/xorq analysis backend#518
paddymul wants to merge 6 commits into
mainfrom
feat/paf-v2-ibis-v2

Conversation

@paddymul

Copy link
Copy Markdown
Collaborator

Summary

  • Fixes IbisAnalysisPipeline to only require ibis (not xorq), adds schema pre-seeding, filters None expressions, adds histogram query support
  • Adds ibis_stats_v2.py with 5 IbisAnalysis subclasses mirroring the pandas/polars stat classes (typing, base summary, numeric, computed, histogram)
  • Adds xorq optional dependency group to pyproject.toml
  • 23 tests using ibis.memtable() — no xorq needed for local testing

Replaces #508 — clean cherry-pick onto current main (after #515, #516, and #517 dependencies merged).

Test plan

  • ruff check passes
  • test_ibis_stats_v2.py — 23 tests using ibis.memtable()
  • CI passes

🤖 Generated with Claude Code

paddymul and others added 2 commits April 30, 2026 08:02
- Fix IbisAnalysisPipeline: remove premature xorq guard (only require ibis),
  add schema pre-seeding for dtype, filter None from expression results,
  add histogram_query_fns support
- Add ibis_stats_v2.py with IbisTypingStats, IbisBaseSummaryStats,
  IbisNumericStats, IbisComputedSummaryStats, IbisHistogramStats
- Add xorq optional dependency group to pyproject.toml
- Add 23 tests using ibis.memtable (no xorq needed for local tests)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Pre-push hook required ruff format on the new ibis files; regenerated
the lock so xorq picks up 0.3.21 (was 0.3.10).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Apr 30, 2026

Copy link
Copy Markdown
Contributor

📦 TestPyPI package published

pip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.13.5.dev25165205118

or with uv:

uv pip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.13.5.dev25165205118

MCP server for Claude Code

claude mcp add buckaroo-table -- uvx --from "buckaroo[mcp]==0.13.5.dev25165205118" --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo-table

📖 Docs preview

🎨 Storybook preview

Replaces the existing v1-style class tests with tests targeting the
yet-to-exist IbisStatPipeline + @stat-based ibis_stats_v2 module.

Adds coverage that was missing in v1:
  - Live histogram (numeric + categorical) — was dead code
  - Structured error capture: a bad @stat surfaces as a StatError,
    not a silent stat dropout
  - DAG validation at pipeline construction
  - Constant-column histogram edge case

Will fail collection until ibis_stat_pipeline.py exists. Implementation
in the next commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the v1-style IbisAnalysis class hierarchy with @stat-decorated
functions executed through a new IbisStatPipeline that mirrors the
typed-DAG semantics of StatPipeline.

Two-phase execution:
  1. Batch aggregate — every @stat with an IbisColumn parameter is
     folded into a single table.aggregate() query and executed once.
  2. Per-column post-batch — computed stats and IbisTable-param stats
     (histograms) run through the standard typed-DAG executor.

Resolves the issues called out in the adversarial review:

  - Histogram is live (was dead code in v1: IbisHistogramStats was
    defined but never added to IBIS_ANALYSIS). Numeric and categorical
    paths now have test coverage that asserts populated histograms.

  - Structured error capture replaces the three `except Exception:
    continue` blocks. Failures inside an @stat or in the batch query
    surface as StatError entries; downstream computed stats still see
    UpstreamError propagation. Tests assert that an intentionally bad
    @stat shows up in the returned errors list.

  - DAG ordering is implicit. requires_summary is no longer a comment —
    parameter names ARE the dependency declaration, validated at
    construction via build_typed_dag.

  - Naming matches the v2 pattern: filename stays ibis_stats_v2 but the
    contents now match the @stat / TypedDict style of pd_stats_v2.

  - Empty-table length=0 fix: the previous `COUNT(c) + SUM(...IS NULL)`
    expression returned NULL on an empty table because SUM-over-empty
    is NULL in SQL. Coalesced to 0.

Adds IbisColumn / IbisTable marker types to stat_func.RAW_MARKER_TYPES
and a small IbisTable injection branch in _execute_stat_func so the
v2 pipeline can pass the table through to histogram-style stats.

Deletes the v1 IbisAnalysisPipeline / IbisAnalysis (no remaining
consumers in buckaroo/ or tests/).

25/25 ibis tests pass; full unit suite passes (modulo the pre-existing
unrelated MCP uvx flake on main).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Asserts the DfStats-shaped wrapper that lets DataFlow / BuckarooWidget
consume an ibis.Table: .sdf, .errs (v1 ErrDict shape), the
verify_analysis_objects classmethod, and .ap.ordered_a_objs.

Will fail collection until IbisDfStatsV2 is added in df_stats_v2.py.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Makes the ibis pipeline reachable from the rest of buckaroo:

  - Adds IbisDfStatsV2 in df_stats_v2.py mirroring DfStatsV2 /
    PlDfStatsV2 (.sdf, .errs, .ap.ordered_a_objs,
    verify_analysis_objects). Any DataFlow consumer that takes an
    arbitrary DFStatsClass now also accepts the ibis path.

  - Adds process_table_v1_compat to IbisStatPipeline so the wrapper
    returns the same {(col, stat): (Exception, kls)} ErrDict shape
    that AnalysisPipeline produced.

  - Adds ordered_a_objs property to IbisStatPipeline so the
    add_analysis path matches StatPipeline.

ibis is still imported lazily inside IbisDfStatsV2 — the rest of the
df_stats_v2 module stays ibis-free.

End-to-end test asserts the round-trip (table -> .sdf with length=5
and mean=3.0) and that v1 errs shape matches the existing convention.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@paddymul

Copy link
Copy Markdown
Collaborator Author

Closing — GitHub UI was showing a phantom 499-file diff after the rebase, even though the actual diff is 10 files (verified via API and gh pr diff). Reopening as a fresh PR from the same branch tip.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant